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Introduction 

Since the publication of the Standards for Technological Literacy in 2000 
(ITEA), there have been a number of new programs developed that are designed 
to teach pre-engineering. Project Lead the Way is one such program. Project 
Lead the Way boasts serving over 1250 schools in 44 states and teaching over 
160,000 students (McVearry, 2003). Efforts are also being made to infuse 
engineering design into technology education programs. One example of this is 
the work of the National Center for Engineering and Technology Education 
(NCETE) partnering with high school technology educators in summer in- 
service workshops to help teachers develop activities and curriculum to instill 
engineering design into technology education programs. According to Douglas, 
Iversen, & Kalyandurg (2004), the engineering community has identified the 
need for teaching engineering in K-12, and this has been supported by the 
American Society of Engineering Education ( ASEE). The ASEE research 
analyzed the current practices of K-12 engineering education. The study stated: 
Clearly, there is a societal argument for the need for engineering education in 
our K-12 classrooms, as technical literacy promotes economic advancement. 

There is a statistical argument, as the number of students entering engineering 
schools declines, related to overall enrollment, and the number of women and 
underrepresented minorities in engineering remains well below the national 
average for higher education (Douglas, Iversen, & Kalyandurg, 2004, p. 3). 

The engineering education community has identified the important role K- 
12 education plays in the success of post-secondary engineering education. 
Teaching engineering content in technology education programs has become a 
recent popular trend with curriculum initiatives such as Project Lead the Way, 
but some states, like New York, have had a course called “Principles of 
Engineering” since the late 1980s (Lewis, 2005). Teaching engineering design 
in K-12 might possibly be good for post-secondary engineering education, but 
does it produce technological problem solvers who have the ability to properly 
manage an ill-defined problem and develop viable solutions? 


Todd R. Kelley (kelley30@uga.edu) is an Assistant Professor in the Department of Industrial 
Technology at Purdue University, West Lafayette, Indiana. 
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Understanding the cognitive strategies of technical problem solvers is 
critical to developing curriculum that develops technologically literate 
individuals. The Standards for Technological Literacy (ITEA, 2000) identified 
the important role of cognition in design by stating: 

To become literate in the design process requires acquiring the cognitive and 
procedural knowledge needed to create a design, in addition to familiarity with 
the processes by which a design will be carried out to make a product or 
system (ITEA, 2000, p. 90). 

Roberts emphasized “the purpose of teaching design is not to bring about 
change in the made world, but change in the student's cognitive skills” (1994, p. 
172). Furthermore, ill-defined problems are more difficult to solve since they 
require more cognitive operations than simpler, well-defined problems 
(Jonassen, 2000). Johnson (1992) suggested a framework for technology 
education curricula, which emphasizes intelligent processes. “Students should 
acquire a repertoire of cognitive and metacognitive skills and strategies that can 
be used when engaged in technological activity such as problem solving, 
decision making, and inquiry” (Johnson, 1992, p. 30). Cognitive and 
metacognitive skills are important thinking processes required for problem 
solving, and these skills should be taught to students in technology education 
courses. Careful examination of the cognitive processes employed by students 
as they work through an ill-defined technical problem provides a means of 
evaluating the effectiveness of a curriculum approach designed to develop 
effective problem solvers. 

Clearly, engineering-focused programs using a classic engineering design 
process model approach the design process differently than technology 
education programs using the design process featured in the Standards for 
Technological Literacy (Hailey, Erekson, Becker, and Thomas, 2005). The most 
notable difference in the design process is that engineering design uses analysis 
and optimization for the mathematical prediction of design solutions. In 
contrast, the technology design process emphasizes selecting a design idea, 
testing the idea through model building, and making final design decisions 
based upon a trial and error process. These vast differences in the approaches to 
design causes one to wonder if students from these technology education 
approaches to design instruction will be able to solve ill-defined problems using 
an engineering design process. Moreover, although both PLTW and the NCETE 
seek to develop engineering-focused design, the purposes of these programs are 
different. Consequently, so are their approaches. While Project Lead the Way 
(Project Lead the Way, 2006) is described as a pre-engineering program, the 
National Center for Engineering and Technology Education seeks to develop 
activities to infuse engineering design into technology education (Hailey, et al., 
2005). Both engineering-focused approaches to design instruction seek to 
provide students with a systematic problem solving method through the 
application of the engineering design process, but will high school students from 
these two different groups perform differently when solving the same ill-defined 


- 51 - 




Journal of Technology Education 


Vol. 1 9 No. 2, Spring 2008 


problem? The purpose of this research was to determine if these two different 
approaches to engineering-focused design instruction will affect how students 
solve ill-defined problems. 


Research Questions 

This research study examined the cognitive processes employed by students 
participating in two different engineering-focused curricular approaches to 
design and problem solving. The following research questions guided the study: 

1. Are students in the selected programs (NCETE & PLTW) using similar 
cognitive processes as they solve ill-defined problems? 

2. Will students in the selected programs (NCETE & PLTW) perform 
similarly when presented with the same ill-defined problem to solve? 

3. What cognitive processes are missing from students participating in the two 
different programs (NCETE & PLTW) and how does each group differ? 

4. Are there important cognitive processes missing from students’ 
performances in both groups (NCETE & PLTW)? 

It is critical to closely examine these important questions as the field of 
technology education considers engineering design as a focus alongside the 
need for developing technological literacy in K-12 learners, a notion supported 
by leaders in the field of technology education (Daugherty, 2005; Lewis, 2004; 
Wicklein, 2006). This research examined how a high school student who has 
learned engineering design solves an assigned ill-defined technical problem. 
This insight can be helpful to develop further curriculum in technology 
education that will develop individuals who are technologically literate and 
effective problem solvers. Another benefit of this study is to gain insight into 
how a high school student, who has learned engineering design methods, 
manages cognitive processes as he or she engages in problem solving when 
confronted with a time constraint. Finally, it is beneficial to identify where 
students fail to properly manage cognitive strategies and to identify what 
cognitive strategies are not utilized in the problem solving process. 

Participants 

This research study examined students participating in two different 
engineering-focused design instruction: Project Lead the Way and a technology 
education program seeking to impart engineering design (NCETE partner). For 
the latter group, four participants were drawn from programs of a participating 
teacher in NCETE in-service workshops conducted at North Carolina A&T 
University. Three subjects were selected from Project Lead the Way schools by 
recommendation from North Carolina A&T NCETE partners. The Project Lead 
the Way participants completed the course Principles of Engineering and were 
currently enrolled in the capstone course titled Engineering Design and 
Development, which is typically taught to seniors in high school. The 
participants selected from a technology education high school program not 
using Project Lead the Way curriculum were students who were taught by an 
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instructor who had benefited from the NCETE in-service teacher workshops 
during the summer of 2006. The participants from both groups were selected by 
their instructors for their problem solving abilities and willingness to participate 
in the study. It is important to note that the NCETE partnered school was 
currently generating new curriculum with a focus on engineering design which 
is why many course titles may not appear to reflect an engineering design focus; 
see Appendix B (available online at scholar.lib.vt.edu/ejournals/JTE/). The 
researcher selected participants for both groups who were homogeneous in 
educational background by requiring the same criteria for the prerequisites of 
mathematics and science as defined by the Project Lead the Way program 
(Project Lead the Way, 2006). The researcher conducted the study near the end 
of the semester so the participants gained as much training on engineering 
design as possible. Demographic information for the participants can be found 
in Appendix B & C. General demographic information about the instructors, 
curriculum, class size, and course titles can be found in Appendix D & E. (all 
appendices available online at scholar.lib.vt.edu/ejournals/JTE/) 

Methodology 

This study compared the cognitive processes used by the participants from 
the two curricular approaches to technology education as they used a design 
process to work through an ill-defined technical problem. The same ill-defined 
technical problem was presented to all the participants. Each participant was 
asked to carefully read the technical problem, identify all constraints he or she 
imposed on the problem, and then asked to begin to develop a solution. Each 
participant worked in isolation from other participants or classmates. The study 
used a “think-aloud” protocol method used in similar studies (Ericsson & 

Simon, 1993; Kruger & Cross, 2001; van Someren, van de Velde, & Sandberg, 
1994). Atman & Bursic (1998) suggested that using a verbal protocol analysis 
for assessing cognitive processes of engineering students is a powerful method 
to understand the process students take when developing a design solution. 
Atman and Bursic stated: “analysis of a verbal protocol enables us to look at a 
subject’s process in detail rather than simply ‘grading’ a final solution. That is, 
we can now grade the ‘process’ as well as the final design” (Atman & Bursic, 
1998, p. 130). Moreover, verbal protocol analysis has been endorsed as a sound 
method for capturing and assessing engineering student’s design processes 
(Atman & Bursic, 1998). Consequently, the participants were asked to verbalize 
their thoughts as they worked through the ill-defined problem. The researcher 
prompted participants to keep talking through the problem when he or she 
stopped verbalizing his or her thoughts; beyond this, the researcher did not 
interact with the participants. The participants were given a total of 30 minutes 
to work through the early stages of the engineering design process; however, 
several participants’ sessions did not use the entire time. Although this time 
constraint limited engagement in the engineering design process, it was 
adequate to study how the student framed the problem and began to develop an 
initial design plan. The data collection included frequency and duration of time 
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of the various mental processes allowing the researcher to break coding data 
into units of time including time on code, total time on each code, percentage of 
time, and total time of the testing session. This method of organizing data by 
time has been used in similar problem solving studies (Welch, 1999). Frequency 
was also recorded, tallying each iteration of the cognitive strategy used by the 
participant. Group mean scores were computed and reported for all cognitive 
processes used for both groups (see Tables 3 & 4). 

The open-ended problem that was given to the participants described 
typical conditions in underdeveloped areas of the world where the domestic 
water is often transported by women and girls. This activity often causes 
physical stress on these women and children, resulting in acute medical 
conditions. The problem statement provided some general information about 
current constraints on this problem as well as solutions that are currently being 
employed. The statement asked the participants to provide details about how 
they would proceed to develop strategies to improve the current conditions in 
these underdeveloped areas. The participants were asked to list all constraints 
that they imposed on the problem. The problem that the participants were asked 
to solve is presented in Figure 1 . 

Framing the Problem 

This study only examined the early stages of the design process. Certainly 
in the time constraint of thirty minutes, a student was unlikely to reach the final 
stages of the design process; therefore he or she was also unlikely to employ all 
of Flalfin’s (1973) mental processes. However, one of the most important stages 
of the engineering design process occurs at the onset of being presented with a 
technical problem: ‘framing the problem’ is this important stage. Experts in the 
field of design identify that framing the problem is a critical step to the design 
process and occurs as soon as the designer is presented with a technical problem 
(Dym, Agogino, Eris, Frey, & Leifer, 2005; SchOn, 1983). This early stage of 
the engineering design process often finds engineers seeking to locate the 
problem space where the search for the solution begins, starting conditions are 
identified, and goals are stated. This problem space creates a partial structure 
from which a solution space can be formed. The solution space structure begins 
to be developed as ideas are generated; this structure is transferred back to 
problem space to again consider solution implications. This method seeks to 
generate cohesion of problem and solution (Cross, 2004). 

Data Gathering and Analysis 

The participants were videotaped for further analysis by the researcher. The 
tape was used to record each participant’s voice as he or she verbalizes their 
thoughts, as well as to record any actions such as sketching, measuring, or any 
other non-verbal cues. Cross (2004) indicated that one weakness of the ‘think 
aloud' verbal protocol method was that it was extremely weak at capturing non- 
verbal thought processes, using observation in combination with the ‘think 
aloud' method was employed to help capture non-verbal cues. This technique of 
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Problem 

In certain underdeveloped areas of the world the majority, if not all, of domestic 
water is transported by women and young girls, causing considerable physical stress and 
resulting in medical conditions that are particularly acute during child-bearing and birth. 
Small villages are scattered throughout rural areas of the world where this has become a 
major issue, in part due to the steep mountainous terrain. 

Currently, water is typically held in plastic or metal vessels and carried in the arms, 
balanced on the head, or attached to the ends of a rod and carried across the shoulders. 
Families who can afford beasts of burden (mules, camels, cattle, etc) employ them in this 
activity, although this is the exception. 

Cultural and political constraints often hinder installation of modern water 
management systems; therefore temporary measures are needed to improve current 
conditions. 

Your Task: 

Describe how you would proceed from this problem statement in order to improve 
the current condition in these underdeveloped areas. Please list all constraints that you 
impose on this problem. As you work through this problem, ‘think aloud’ your strategies 
for deriving a solution. 


Figure 1. The ill-defined problem used in the study. 

combining a verbal protocol with a video of the testing session is known as 
observational protocol and is a data collection method used to assess student 
design and problem solving strategies (Laeser, Moskal, Knecht, & Lasich, 
2003). The data collection included frequency and duration of time of the 
various mental processes. 

This research study focused on cognitive processes from a list of 17 mental 
processes that were identified by Halfin ( 1973). Halfin used writings from ten 
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high-level designers including Buckminster Fuller, Thomas Edison, and Frank 
Lloyd Wright. Halfin used a Delphi technique to identify mental processes that 
were universal for these expert engineers and designers. Hill (1997) developed a 
computer analysis tool called the Observation -Procedure for Technology 
Education Mental Processes (OPTEMP) to assess problem-solving activities in 
technology education by employing Halfin's code of mental processes. The 
study herein used a revised and updated OPTEMP computer program to assist in 
coding and recording the frequency and duration of time of the cognitive 
processes employed by students as they worked through the selected ill-defined 
technical problem. The researcher coded the actions and cognitive processes 
used by each participant as he or she worked through the technical problem. The 
number of frequencies and the time spent on each strategy were compiled and a 
total was recorded in the OPTEMP output. 

Microsoft Excel software was used to process the data files generated by 
the OPTEMP program. Careful analysis of the percentage of time and frequency 
spent on the various cognitive strategies provided insight into mental processes 
employed by the students as they worked to frame the ill-defined problem as 
well as a comparison of group means scores. 

Table 1 

Halfin’s (1973) Original Cognitive Processes 


Mental Methods 

Code 

Analyzing 

AN 

Communicating 

CM 

Computing 

CO 

Creating 

CR 

Defining problem) s) 

DF 

Designing 

DE 

Experimenting 

EX 

Interpreting data 

ID 

Managing 

MA 

Measuring 

ME 

Modeling 

MO 

Models/prototypes 

MP 

Observing 

OB 

Predicting 

PR 

Questions/hypotheses 

QH 

Testing 

TE 

Visualizing 

VI 


Findings 

Although a thirty-minute or shorter examination is inadequate in 
understanding the entire process taken by problem solvers, it can provide great 
insight into an individual’s ability to organize the problem, constraints, and 
criteria in order to begin developing a solution. Importantly, the reader is 
reminded that the findings of this study are very limited in their generalizability. 
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Are students in these different programs using similar cognitive processes as 
they solve ill-defined problems? 

The research revealed that both groups used similar cognitive strategies as 
they worked though the ill-defined problem. Both groups employed at least six 
of the ten mental processes that were identified in the test sessions. The 
cognitive strategy analysis (AN) was the most common mental processes 
employed. This code was recorded when the researcher witnessed the 
participant breaking down the problem and identifying constraints and criteria. 
The participants spent from 19 to 54 percent of their time doing this. The group 
mean was 10.70 minutes for the PLTW group and 7.42 minutes for the NCETE 
group. The duration of time that the two groups spent on the various strategies 
varied considerably (See Tables 2 -4). 

Will students in these programs perform similarly when presented with the same 
ill-defined problem to solve ? 

The results of this research revealed that the two groups did perform 
differently with respect to time spent developing solutions (coded DE). Often 
this mental process is considered the most critical in determining how an 
individual designs a solution. Kruger and Cross (2001) proposed that designers 
are either solution driven or problem driven. Welch and Lim (2000) have noted 
that novice designers often become stuck in the problem space and fail to 
generate solutions. The results of this study reveal that group NCETE partner 
group spent more time generating solutions than the PLTW group. The NCETE 
group spent from 18 to 32 percent of their time designing and talking about 
solution ideas. In contrast, the PLTW group only spent from 3 to 8 percent 
dialoging design solutions. Comparing the group means, the NCETE group 
spent an average of 5.40 minutes generating design solutions in contrast to an 
average of 1 .77 minutes spent by the PLTW group. Although creative designers 
are known for generating multiple solutions, there is a danger in generating 
solutions too quickly due to an incomplete understanding of the problems 
(Welch, 1999). It is important to consider that while the NCETE group spent 
more time generating solutions, the PLTW group spent a considerable amount 
of time defining and analyzing the problem. Comparatively, architects are 
problem solvers who generate multiple solutions to design problems, whereas 
engineers are often trained to locate a single solution that works in a timely and 
cost effective manner (Akin, 2001). Although participant number six developed 
only one design idea, eight frequency counts are reported (Table 2) and 
represent discussions of a single design idea. Participant number six was 
convinced that the idea was the best solution, possibly based on his knowledge 
of similar cultures who have struggled with this problem. Ball, Ormerod, & 
Motley (2004) refer to this approach to solving problems as “case-driven” and 
refer to it as a novice designer approach. The case-driven approach is used to 
quickly move to a solution by recognizing the similarity of the current problem 
to a problem encountered in the past and to apply a solution from the earlier 


- 57 - 




Journal of Technology Education 


Vol. 1 9 No. 2, Spring 2008 




H 

IT) 

r- 

t-" 

q 

q 

so 

o 

CM 

r- 

O 

o 




— H 

00 



co 


o 

08 

vo 

O 

o 



d 

q 

OO 

so 

08 

d 

d 


o 

d 

t_h 


— f 


<N 

CO 











=tfc 














■** 














S3 

CQ 

H 

On 

>n 

Os 

in 

so 

q 

o 

i— H 

o 

o 

vo 

S' 



q 

q 

>n 

>n 

CO 

o 


q 



CM 

s 

s 

'p 


IT) 

oo 

q 

,_H 

CM 

d 


CO 

d 


«n 

CM 

o 

u 

a 













"o 

c- 















k, 

CM 

CO 

CO 

i-H 

o 


o 

o 

— H 

o 

i-H 

-s; 



CM 

vo 

CO 

T— 1 

CM 



CM 



o 















to 










































g 


H 

ON 

SO 

o 

oo 

CM 

q 

o 

o 

o 

o 


5*. 



00 

oo 

o 

’-H 

t> 

co 

o 

o 

o 

o 


<3 


CM 

d 

s 

CO 

CM 

CO 

d 

o 

d 

d 

o 

co 

4fc 



q 

CM 








o 














1 

C 

a 

& 

H 

oo 



Os 

VO 


o 

o 

o 

o 

vo 


>n 

q 

CO 

CO 

>n 

q 





CM 



i— H 

in 

CO 

d 

H 

d 





CM 















-S 

t- 













.g 

a 

Oh 













s 



q 

<N 

2 

H 

VO 

CM 

o 

o 

o 

o 

08 





CM 









q 

'I 














>3 




























>3 


H 

SO 

<N 

CO 

o 

o 

o 

CM 

o 

o 

o 

o 

50 

CJ 



CM 

ON 

CO 

>n 

oo 

o 

d 

VO 

l> 

CO 

08 

08 

CO 

o 

o 

o 

d 

o 

d 

o 

e 

co 



q 










a. 

4t 



























si 

.go 

S 

CQ 

H 

O 

r- 

o 

o 

_ 

so 

oo 

o 

o 

o 

CM 

# a 


CO 

CO 

q 


q 

in 

q 




in 

Q 

*3 

u 


in 

- 

in 


CM 

CM 





S 

CM 


C5 













S 

a. 













K 



vo 

q 

o 

o 

00 

CM 


o 

o 

o 

i-H 

1 

R .1 




CO 

CM 








08 


H 

r- 

SO 

in 

VO 


o 

vo 

vo 

o 

CM 

o 

Q O 5 



q 

<N 

>n 

oo 

q 

d 

q 

oo 

CO 

o 

d 

q 

CM 

so. 

q 

o 

o 

q 

o 

^ ~ 

=H= 


<N 


CO 









•~ £ 
ts ^ 

C 

a 

a 

H 

CM 

CO 

oo 

O 

O 

o 

OO 

CO 

o 

o 

oo 



CM 

CM 

>n 

CM 

CO 


n 

q 


q 

q 

§> 

Co“^ 

p 


SO 

>n 

oo 

CM 

d 


d 

q 


d 

oo 

CM 

.§ .S 

hs ^ 

u 

a 

Oh 

k, 

in 

CO 

CO 

so 

q 

o 

vo 

CM 

o 


o 



y—t 

CO 

q 

y—i 




y—t 



CO 

^ Jl 













y—i 

si ^ 
Q 




























Sj 

o s; 


t/3 












s; 














^ s 














s ^ 

v *- 1 

lT 


Halfi 

Code 

DF 

AN 

DE 

MA 

PR 

X 

O 

CM 

MO 

CO 

g 

Total 




1 












- 58 - 




Journal of Technology Education 


Vol. 1 9 No. 2, Spring 2008 


problem. Conversely, Cross (2004) suggested that expert problem solvers with 
experience in designing move quickly from the problem frame to proposing a 
solution. Considering that this participant spent a great deal of time identifying 
the constraints and criteria (analysis) and very little time simply defining the 
problem, he may be demonstrating his ability to design quickly and efficiently 
as opposed to lacking creative idea generation (See Table 3). 

What cognitive processes are missing from students representing the two 
different programs, and how does each group differ ? 

Of Halfin’s 17 mental processes, seven processes were never employed by 
either group. A close examination of the seven missing processes resulted in a a 
logical explanation for most of them. For example, models/prototypes (code 
MP) were never employed, quite possibly due to the limited time constraints and 
lack of available modeling materials. Actually, use of models and prototypes 
was not expected by the researcher at this stage of the design process. 
Interpreting data (ID) was not often employed by participants (only one 
participant used it to a very limited extent) in this study. This is likely due to the 
fact that there were little data to interpret from the problem statement. 

Measuring (ME) was a mental process that could be applied to this ill- 
defined problem if a heuristic (as suggested by Koen, 2003) was applied to the 
constraints presented in the problem. However, none of the participants 
employed this strategy. Measuring, as defined by Halfin is “the process of 
describing characteristics (by the use of numbers) of a phenomenon problem, 
opportunity, element, object, event, system, or point of view in terms, which are 
transferable” ( 1973). Considering that a major distinction between the 
technology and engineering design processes is that engineering design applies 
mathematical prediction and optimization, this missing cognitive process is 
significant. The absence of this cognitive strategy causes one to speculate 
whether or not students in an engineering-focused design program have any 
increased ability or need to use mathematics to predict design solution compared 
to students from technology education programs without an engineering design 
focus, at least with respect to solving an ill-defined problem. Thus, this study 
does not support the notion that students in an engineering-focused program 
apply mathematical prediction and optimization in their problem solving. The 
other missing cognitive processes from both groups included creating (CR), 
experimenting (EX), observing (OB), testing (TE) and visualizing (VI). 

Are there important cognitive processes missing from students’ performances in 
both groups? 

As mentioned above, measuring (ME) was never utilized by any participant 
in the study. Computing (CO) was only used by two participants, one from each 
group applied a quantity to estimate potential distances traveled or the altitude 
of the mountain terrain. However, no participants used estimations to predict the 
results of design solutions. This has been identified as a missing piece in the 
technological design process (Hailey, et al., 2005; Wicklein, 2006). The 
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Table 3 

Frequency and Time Spent in Halfin ’s Mental Design Processes within the 
PLTW School Group (f = frequency, T = time, %T = percent of time) 


% 


Halfin' 

Code 

F 

Participant #5 
T %T 

Participant #6 
/ T %T 

/ 

Participant #7 
T %T 

DF 

8 

2.56 

9.02 

9 

2.17 

18.08 

38 

7.24 

27.23 

AN 

168 

13.39 

47.16 

55 

4.53 

37.75 

91 

14.18 

53.33 

DE 

22 

2.56 

9.02 

8 

0.40 

3.33 

19 

2.34 

8.80 

MA 

2 

0.16 

0.56 

12 

1.57 

13.08 

11 

1.46 

5.49 

PR 

33 

6.05 

21.31 

17 

2.10 

17.50 

11 

1.24 

4.66 

QH 

0 

0 

0.00 

1 

0.13 

1.08 

1 

0.13 

0.49 

CM 

0 

0 

0.00 

1 

0.7 

5.83 

0 

0 

0.00 

MO 

13 

3.11 

10.95 

0 

0 

0.00 

0 

0 

0.00 

CO 

3 

0.16 

0.56 

0 

0 

0.00 

0 

0 

0.00 

ID 

0 

0 

0.00 

0 

0 

0.00 

0 

0 

0.00 

Total 

247 

28.39 

100.00 

103 

12.00 

100.00 

171 

26.59 

100.00 


Table 4 

Comparison of Times and Frequencies for PLTW and NCETE Participants by 


Halpin 's Categories 



NCETE 

Group 

Frequency 

PLTW 

Group 

NCETE 

Group 

Time 

PLTW 

Group 

DF 

14.25 

18.33 

4.55 

3.99 

AN 

38.00 

104.67 

7.42 

10.70 

DE 

27.50 

16.33 

5.40 

1.77 

MA 

7.00 

8.33 

1.05 

1.06 

PR 

9.5 

20.33 

1.60 

3.13 

QH 

3.75 

0.67 

0.75 

0.09 

CM 

1.75 

0.33 

0.42 

0.23 

MO 

8.00 

4.33 

1.79 

1.04 

CO 

0.25 

1.00 

0.04 

0.05 

ID 

0.25 

0.00 

0.10 

0.00 

Total 

110.25 

173.67 

23.31 

22.33 


minimal use of this cognitive strategy should be a concern for those who believe 
students in engineering related programs have the ability to apply their math 
skills to predict design solutions. 


Reliability 

The measure of inter-coder reliability revealed a high degree of 
consistency. Two researchers independently coded 10 % of four of the seven 
protocols as outlined by Evans (1995). Segments were selected at the beginning, 
middle, and at the end of the assessed protocols to ensure that the reliability 
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check was conducted at various stages of the testing session. The total times that 
each coder ascribed to Halfin's mental processes are presented in Table 5. 
Standard deviations ranged from .523 for Analysis to .092 for Managing and 
Predicting. 


Table 5 


Inter-coder Reliability Agreement Results 


Halpin Category 

Coder #1 

Time 
Coder # 2 

Standard Deviation 

DF (Defining the Problem) 

4.41 

4.53 

0.085 

AN (Analysis) 

4.05 

3.31 

0.523 

DE (Designing) 

0.46 

1.01 

0.389 

MA (Managing) 

0.00 

0.13 

0.092 

QH (Questioning) 

0.21 

0.15 

0.042 

CM (Communicating) 

0.18 

0.34 

0.113 

PR (Predicting) 

0.13 

0.00 

0.092 

Total Time 

9.44 

9.47 



Discussion 

As the field of technology education has been moving to include 
engineering, a variety of new curriculum projects have emerged. Some 
examples of curriculum projects include Project Lead the Way, and ITEA’s 
Engineering by Design, Engineering the Future, and Engineering is Elementary. 
As these engineering oriented programs are implemented into schools and new 
curriculum is implemented, it is important to evaluate their effectiveness in 
increasing students’ cognitive abilities with respect to problem solving. One 
way to do this is to examine students as they work to solve ill-defined problems. 
The method used in this study can provide a heightened awareness of what is 
really happening in the minds of the students as they work to solve a problem. 
Technology education programs have often emphasized design and problem 
solving (Flowers, 1998; Foster, 1994; Plaza, 2004), but little research has been 
done to determine how effective these activities are in developing skills, skilled 
problem solvers, and excellent designers (Lewis, 1999). More research needs to 
be conducted in technology education to examine the cognitive capabilities of 
students and observational protocols are a sound methodology that is cost 
effective. According to the results of this study, students do perform differently 
with respect to solving ill-defined problems when grouped by engineering- 
focused programs. Additional research should be done to extend the results of 
this study by increasing the sample size and expand the sample to include other 
technology education programs with and without an engineering focus. It is 
critical for the field of technology education to consider the characteristics and 
outcomes it would like to develop in its students. Among these outcomes are 
students who are creative problem solvers who can generate multiple solutions 
on the one hand or problem solvers who can quickly locate the most efficient 
and cost effective solution on the other hand. Certainly, a case can be made for 
both types of problem solvers, quite possibly a blend of experiences in problem 
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solving would be appropriate for the field to consider as the integration of 
engineering design continues. 
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